Camera Module

ABSTRACT

An image processor for a camera module having an image sensor that captures an image for transfer to a printer module of a compact printer system including a capture unit that captures the image from the image sensor and transforms it to planarized linear RGB form. The image processor also includes one or more interface units communicating with components of said camera module under control of the central processing unit and image processing units, being an application specific integrated circuits, in communication with said image storage memory to transform said image for transmission. A Serial Bus interface communicates with the Serial Bus of a compact printer system to transfer data and power between modules. In one application, the camera module captures an image, which is transferred to the image storage memory of a printer module for printing by a printhead.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.10/636,216 filed on Aug. 8, 2003, which is a divisional of U.S.application Ser. No. 09/575,157 filed on May 23, 2000 (now abandoned),there content of which is incorporated herein by reference.

FIELD OF THE INVENTION

The invention relates to a compact printer system able to printfull-color, business card size documents from a device about the size ofa pen. The system includes various hot-connectable modules that providea range of functions, including a camera module.

Reference may be had to co-pending applications and patent claimingpriority from Australian Provisional Patent Application number PQ0560dated 25 May 1999. The co-pending applications and patent describerelated modules and methods for implementing the compact printer system.The co-pending applications and patents are as follows:

US OFFICIAL NO. TITLE 6,924,907 Compact Color Printer Module 6,712,452Modular Compact Printer System 6,416,160 Nozzle Capping Mechanism6,238,043 Ink Cartridge for Compact Printer System 6,958,826 Controllerfor Printer Module 6,812,972 Camera Module for Compact Printer System6,553,459 Memory Module for Compact Printer System 6,967,741 EffectsModule for Compact Printer System 6,956,669 Effects Processor forEffects Module 6,903,766 Timer Module for Compact Printer System6,804,026 Color Conversion Method for Compact Printer System 7,259,889Method and Apparatus of Dithering 6,975,429 Method and Apparatus ofImage Conversion

BACKGROUND OF THE INVENTION

Microelectronic manufacturing techniques have led to the miniaturizationof numerous devices. Mobile phones, personal digital assistant devices,and digital cameras are very common examples of the miniaturizationtrend.

One device that has not seen the advantage of microelectronicmanufacturing techniques is the printer. Commercially available printersare large compared to many of the devices they could support. Forinstance, it is impractical to carry a color printer for the purpose ofinstantly printing photographs taken with known compact digital cameras.

A compact printhead has been described in United States patents andhereby incorporated by cross reference:

U.S. Pat. No. TITLE 6,428,133 Ink jet printhead having a moving nozzlewith an externally arranged actuator 6,526,658 Method of manufacture ofan ink jet printhead having a moving nozzle with an externally arrangedactuator 6,390,591 Nozzle guard for an ink jet printhead 7,018,016Fluidic seal for an ink jet nozzle assembly 6,328,417 Ink jet printheadnozzle array

In order to construct a camera module for a compact printer system it isnecessary to address the problem of control of the image capture andtransformation process. Methods of very large scale integration andmicroelectronic manufacturing are known but have not been applied to theneeds of a compact printer system. No suitable image processor existsfor a camera module that connects directly to a printer module.

SUMMARY OF THE INVENTION

According to an aspect of the present invention there is provided acamera module comprising:

an image sensor for capturing an image;

image storage memory for storing said image;

image processing units, being application specific integrated circuits,in communication with said image storage memory to transform said imageinto a device independent color space; and

a serial bus interface for communicating said transformed image with aserial bus of a printer module interconnected with the camera module.

Other aspects are also disclosed

Further features of the invention will be evident from the followingdescription.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to assist with describing preferred embodiments of theinvention, reference will be made to the following figures in which:

FIG. 1 is a printer module;

FIG. 2 is a camera module;

FIG. 3 is a memory module;

FIG. 4 is a communication module;

FIG. 5 is a flash module;

FIG. 6 is a timer module;

FIG. 7 is a laser module;

FIG. 8 is an effects module;

FIG. 9 is a characters module;

FIG. 10 is an adaptor module;

FIG. 11 is a pen module;

FIG. 12 is a dispenser module;

FIG. 13 is a first compact printer configuration;

FIG. 14 is a second compact printer configuration;

FIG. 15 is a third compact printer configuration;

FIG. 16 is a fourth compact printer configuration;

FIG. 17 is a block diagram of a controller for the camera module of FIG.2;

FIG. 18 is a block diagram of the image capture unit;

FIG. 19 is a block diagram of the image enhancement unit;

FIG. 20 is a block diagram of the white balance and range expansionprocess;

FIG. 21 is a block diagram of the resample process;

FIG. 22 is a block diagram of the color conversion process;

FIG. 23 is a block diagram of the sharpen process;

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIGS. 1 to 12, there are shown various modules thattogether form a compact printer system. Individual modules can beattached and detached from the compact printer configuration to allow auser-definable solution to business-card sized printing. Images can alsobe transferred from one compact printer to another without the use of asecondary computer system. Modules have a minimal user-interface toallow straightforward interaction.

A compact printer system configuration consists of a number of compactprinter modules connected together. Each compact printer module has afunction that contributes to the overall functionality of the particularcompact printer configuration. Each compact printer module is typicallyshaped like part of a pen, physically connecting with other compactprinter modules to form the complete pen-shaped device. The length ofthe compact printer device depends on the number and type of compactprinter modules connected. The functionality of a compact printerconfiguration depends on the compact printer modules in the givenconfiguration.

The compact printer modules connect both physically and logically. Thephysical connection allows modules to be connected in any order, and thelogical connection is taken care of by the compact printer Serial Bus—abus that provides power, allows the modules to self configure andprovides for the transfer of data.

In terms of physical connection, most compact printer modules consist ofa central body, a male connector at one end, and a female connector atthe other. Since most modules have both a male and female connector, themodules can typically be connected in any order. Certain modules onlyhave a male or a female connector, but this is determined by thefunction of the module. Adaptor modules allow these single-connectormodules to be connected at either end of a given compact printerconfiguration.

A four wire physical connection between all the compact printer modulesprovides the logical connection between them in the form of the compactprinter Serial Bus. The compact printer Serial Bus provides power toeach module, and provides the means by which data is transferred betweenmodules. Importantly, the compact printer Serial Bus and accompanyingprotocol provides the means by which the compact printer systemauto-configures, reducing the user-interface burden on the end-user.

Compact printer modules can be grouped into three types:

-   -   image processing modules including a Printer Module (FIG. 1), a        Camera Module (FIG. 2), and a Memory Module (FIG. 3). Image        processing modules are primarily what sets the compact printer        system apart from other pen-like devices. Image processing        modules capture, print, store or manipulate photographic images;    -   housekeeping modules including an Adapter Module (FIG. 10), an        Effects Module (FIG. 8), a Communications Module (FIG. 4), and a        Timer Module (FIG. 6). Housekeeping modules provide services to        other modules or extended functionality to other modules; and    -   isolated modules including a Pen Module (FIG. 11) and a Laser        Module (FIG. 7). Isolated modules are those that attach to the        compact printer system but are completely independent of any        other module. They do not necessarily require power, and may        even provide their own power. Isolated Modules are defined        because the functionality they provide is typically incorporated        into other pen-like devices.

Although housekeeping modules and isolated modules are useful componentsin a compact printer system, they are extras in a system dedicated toimage processing and photographic manipulation. Life size (1:1)illustrations of the compact printer modules are shown in FIGS. 1 to 12,and example configurations produced by connecting various modulestogether are shown in FIGS. 13 to 16.

FIG. 1 shows a printer module that incorporates a compact printheaddescribed in co-pending United States patent applications listed in theBackground section of this application, incorporated herewith byreference, and referred to herewith as a Memjet printhead. The Memjetprinthead is a drop-on-demand 1600 dpi inkjet printer that producesbi-level dots in up to 4 colors to produce a printed page of aparticular width. Since the printhead prints dots at 1600 dpi, each dotis approximately 22.5 μm in diameter, and spaced 15.875 μm apart.Because the printing is bi-level, the input image should be dithered orerror-diffused for best results. Typically a Memjet printhead for aparticular application is page-width. This enables the printhead to bestationary and allows the paper to move past the printhead. A Memjetprinthead is composed of a number of identical ½ inch Memjet segments.

The printer module 10 comprises a body 11 housing the Memjet printhead.Power is supplied by a three volt battery housed in battery compartment12. The printhead is activated to commence printing when a business card(or similar sized printable media) is inserted into slot 13. Maleconnector 14 and female connector 15 facilitate connection of othermodules to the printer module 10.

FIG. 2 shows a camera module 20. The camera module provides apoint-and-shoot camera component to the compact printer system as ameans of capturing images. The camera module comprises a body 21 havinga female connector 22. A lens 23 directs an image to an image sensor andspecialized image processing chip within the camera 24. A conventionalview finder 25 is provided as well as a lens cap 26. An image iscaptured when the Take button 27 is pushed. Captured images aretransferred to the Printer Module 10 for subsequent printing,manipulation, or storage. The Camera Module also contains a self-timermode similar to that found on regular cameras.

FIG. 3 shows a Memory Module 30 comprising a body 31, LCD 32, IN button33, OUT button 34 and SELECT button 35. The Memory Module 30 is astandard module used for storing photographic images captured by theCamera 20. The memory module stores 48 images, each of which can beaccessed either at full resolution or at thumbnail resolution. Fullresolution provides read and write access to individual images, andthumbnail resolution provides read access to 16 images at once inthumbnail form.

The Memory Module 30 attaches to other modules via a female connector 36or male connector 37. The male and female connectors allow the module tobe connected at either end of a configuration. Power is provided fromthe Printer Module 10 via the Serial Bus.

A Communications Module 40 is shown in FIG. 4. The communications module40 consists of a connector 41 and a cable 42 that terminates in anappropriate connector for a computer port, such as a USB port, RS232serial port or parallel port. The Communications Module 40 allows thecompact printer system to be connected to a computer. When so connected,images can be transferred between the computer and the various modulesof the compact printer system. The communications module allows capturedimages to be downloaded to the computer, and new images for printing tobe uploaded into the printer module 10.

A Flash Module 50 is shown in FIG. 5. The Flash Module 50 is used togenerate a flash with flash cell 51 when taking photographs with theCamera Module 20. The Flash Module attaches to other modules via femaleconnector 52 and male connector 53. It contains its own power source.The Flash Module is automatically selected by the Camera Module whenrequired. A simple switch allows the Flash Module to be explicitlyturned off to maximize battery life.

FIG. 6 shows a Timer Module 60 that is used to automate the taking ofmultiple photos with the Camera Module 20, each photo separated by aspecific time interval. The captured photos are stored in Memory Module30. Any flash requirements are handled by the Camera Module 20, and cantherefore be ignored by the Timer Module. The Timer Module 60 consistsof a body 61 housing a LCD 62, START/STOP button 63 and UNITS button 64.A SELECT button 65 allows the user to select time units and the numberof units are set by UNITS button 64. The Timer Module 60 includes a maleconnector 66 and female connector 67. The Timer Module takes its powerfrom the Printer Module 10 via the Serial Bus.

A Laser Module 70 is shown in FIG. 7. The Laser Module 70 consists of abody 71 containing a conventional laser pointer operated by button 72.As the Laser Module is a terminal module it only has one connector,which in the example is a male connector 73. The Laser Module is anisolated module, in that it does not perform any image capture, storage,or processing. It exists as a functional addition to the compact printersystem. It is provided because laser pointer services are typicallyincorporated into other pen-like devices. The Laser Module contains itsown power supply and does not appear as a device on the Serial Bus.

The Effects Module shown in FIG. 8 is an image processing module. Itallows a user to select a number of effects and applies them to thecurrent image stored in the Printer Module 10. The effects includeborders, clip-art, captions, warps, color changes, and painting styles.The Effects Module comprises a body 81 housing custom electronics and aLCD 82. A CHOOSE button 83 allows a user to choose between a number ofdifferent types of effects. A SELECT button 84 allows the user to selectone effect from the number of effects of the chosen type. Pressing theAPPLY button 85 applies the effect to image stored in the Printer Module10. The Effects Module obtains power from the Serial Bus. Male connector86 and female connector 87 allow the Effects Module to be connected toother compact printer system modules.

FIG. 9 shows a Character Module 90 that is a special type of EffectsModule (described above) that only contains character clip-art effectsof a given topic or genre. Examples include The Simpsons®, Star Wars®,Batman®, and Dilbert® as well as company specific modules for McDonalds®etc. As such it is an image processing module. It consists of a body 91housing custom electronics and a LCD 92. SELECT button 93 allows theuser to choose the effect that is to be applied with APPLY button 94.The Character Module obtains power from the Serial Bus through maleconnector 95 and female connector 96.

The Adaptor Module 100, shown in FIG. 10, is a female/female connectorthat allows connection between two modules that terminate in maleconnectors. A male/male connector (not shown) allows connection betweentwo modules that terminate in female connectors. The Adaptor Module is ahousekeeping module, in that it facilitates the use of other modules,and does not perform any specific processing of its own.

All “through” modules have a male connector at one end, and a femaleconnector at the other end. The modules can therefore be chainedtogether, with each module connected at either end of the chain. Howeversome modules, such as the Laser Module 70, are terminating modules, andtherefore have either a male or female connector only. Suchsingle-connector modules can only be connected at one end of the chain.If two such modules are to be connected at the one time, an AdaptorModule 100 is required.

FIG. 11 shows a Pen Module 110 which is a pen in a module form. It is anisolated module in that it attaches to the compact printer system but iscompletely independent of any other module. It does not consume orrequire any power. The Pen Module is defined because it is a convenientextension of a pen shaped, pen sized device. It may also come with a cap111. The cap may be used to keep terminating connectors clean in thecase where the chain ends with a connector rather than a terminatingmodule.

To assist with accurately feeding a business card sized print media intoslot 13 of the printer module 10, a dispenser module 120 is provided asshown in FIG. 12. The dispenser module 120 comprises a body 121 thatholds a store of business card sized print media. A Printer Module 10locates into socket 122 on the dispenser module 120. When correctlyaligned, a card dispensed from the dispenser module by slider 123 entersslot 13 and is printed.

In the sense that a minimum configuration compact printer system must beable to print out photos, a minimum compact printer configurationcontains at least a Printer Module 10. The Printer Module holds a singlephotographic image that can be printed out via its Memjet printer. Italso contains the 3V battery required to power the compact printersystem.

In this minimum configuration, the user is only able to print outphotos. Each time a user inserts a business card 130 into the slot inthe Printer Module, the image in the Printer Module is printed onto thecard. The same image is printed each time a business card is insertedinto the printer. In this minimum configuration there is no way for auser to change the image that is printed. The dispenser module 120 canbe used to feed cards 130 into the Printer Module with a minimum offuss, as shown in FIG. 13.

By connecting a Camera Module 20 to the minimum configuration compactprinter system the user now has an instant printing digital camera in apen, as shown in FIG. 14. The Camera Module 20 provides the mechanismfor capturing images and the Printer Module 10 provides the mechanismfor printing them out. The battery in the Printer Module provides powerfor both the camera and the printer.

When the user presses the “Take” button 27 on the Camera Module 20, theimage is captured by the camera 24 and transferred to the Printer Module10. Each time a business card is inserted into the printer the capturedimage is printed out. If the user presses “Take” on the Camera Moduleagain, the old image in the Printer Module is replaced by the new image.

If the Camera Module is subsequently detached from the compact printersystem, the captured image remains in the Printer Module, and can beprinted out as many times as desired. The Camera Module is simply thereto capture images to be placed in the Printer Module.

FIG. 15 shows a further configuration in which a Memory Module 30 isconnected to the configuration of FIG. 14. In the embodiment of FIG. 15,the user has the ability to transfer images between the Printer Module10 and a storage area contained in the Memory Module 30. The userselects the image number on the Memory Module, and then either sendsthat image to the Printer Module (replacing whatever image was alreadystored there), or brings the current image from the Printer Module tothe specified image number in the Memory Module. The Memory Module alsoprovides a way of sending sets of thumbnail images to the PrinterModule.

Multiple Memory Modules can be included in a given system, extending thenumber of images that can be stored. A given Memory Module can bedisconnected from one compact printer system and connected to anotherfor subsequent image printing.

With the Camera Module 20 attached to a Memory Module/Printer Modulecompact printer system, as shown in FIG. 15, the user can “Take” animage with the Camera Module, then transfer it to the specified imagenumber in the Memory Module. The captured images can then be printed outin any order.

By connecting a Communications Module 40 to the minimum configurationcompact printer system, the user gains the ability to transfer imagesbetween a PC and the compact printer system. FIG. 16 shows theconfiguration of FIG. 15 with the addition of a Communications Module40. The Communications Module makes the Printer Module 10 and any MemoryModules 30 visible to an external computer system. This allows thedownload or uploading of images. The communications module also allowscomputer control of any connected compact printer modules, such as theCamera Module 20.

In the general case, the Printer Module holds the “current” image, andthe other modules function with respect to this central repository ofthe current image. The Printer Module is therefore the central locationfor image interchange in the compact printer system, and the PrinterModule provides a service to other modules as specified by userinteraction.

A given module may act as an image source. It therefore has the abilityto transfer an image to the Printer Module. A different module may actas an image store. It therefore has the ability to read the image fromthe Printer Module. Some modules act as both image store and imagesource. These modules can both read images from and write images to thePrinter Module's current image.

The standard image type has a single conceptual definition. The imagedefinition is derived from the physical attributes of the printhead usedin the Printer Module. The printhead is 2 inches wide and prints at 1600dpi in cyan, magenta and yellow bi-level dots. Consequently a printedimage from the compact printer system is 3200 bi-level dots wide.

The compact printer system prints on business card sized pages (85 mm×55mm). Since the printhead is 2 inches wide, the business cards areprinted such that 1 line of dots is 2 inches. 2 inches is 50.8 mm,leaving a 2 mm edge on a standard business-card sized page. The lengthof the image is derived from the same card size with a 2 mm edge.Consequently the printed image length is 81 mm, which equals 5100 1600dpi dots. The printed area of a page is therefore 81 mm×51 mm, or5100×3200 dots.

To obtain an integral contone to bi-level ratio a contone resolution of267 ppi (pixels per inch) is chosen. This yields a contone CMY page sizeof 850×534, and a contone to bi-level ratio of 1:6 in each dimension.This ratio of 1:6 provides no perceived loss of quality since the outputimage is bi-level.

The printhead prints dots in cyan, magenta, and yellow ink. The finaloutput to the printed page must therefore be in the gamut of theprinthead and take the attributes of the inks into account. It would atfirst seem reasonable to use the CMY color space to represent images.However, the printer's CMY color space does not have a linear response.This is definitely true of pigmented inks, and partially true fordye-based inks. The individual color profile of a particular device(input and output) can vary considerably. Image capture devices (such asdigital cameras) typically work in RGB (red green blue) color space, andeach sensor will have its own color response characteristics.

Consequently, to allow for accurate conversion, as well as to allow forfuture image sensors, inks, and printers, the CIE L*a*b* color model[CIE, 1986, CIE 15.2 Colorimetry: Technical Report (2^(nd) Edition),Commission Internationale De l'Eclairage] is used for the compactprinter system. L*a*b* is well defined, perceptually linear, and is asuperset of other traditional color spaces (such as CMY, RGB, and HSV).

The Printer Module must therefore be capable of converting L*a*b* imagesto the particular peculiarities of its CMY color space. However, sincethe compact printer system allows for connectivity to PCs, it is quitereasonable to also allow highly accurate color matching between screenand printer to be performed on the PC. However the printer driver or PCprogram must output L*a*b*.

Each pixel of a compact printer image is therefore represented by 24bits: 8 bits each of L*, a*, and b*. The total image size is therefore1,361,700 bytes (850×534×3).

Each image processing module is able to access the image stored in thePrinter Module. The access is either to read the image from the PrinterModule, or to write a new image to the Printer Module.

The communications protocol for image access to the Printer Moduleprovides a choice of internal image organization. Images can be accessedeither as 850×534 or as 534×850. They can also be accessed ininterleaved or planar format. When accessed as interleaved, each pixelin the image is read or written as 24 bits: 8 bits each of L*, a*, b*.When accessed as planar, each of the color planes can be read or writtenindependently. The entire image of L* pixels, a* pixels or b* pixels canbe read or written at a time.

As mentioned above, a camera, such as the camera module 20, acquires animage in RGB format but the printer module 10 prints the image in CMYformat. The camera module includes an application specific integratedcircuit configured as an image processor to collect an image from a CMOSimage sensor and transforms the image to a format suitable fortransmission to the image processor of the print module. FIG. 17 shows aschematic of the image processor 170. Each element of the controller isdescribed in detail below together with examples of operation of theelements.

The image processor 170 is designed to be fabricated using a 0.25 micronCMOS process, with approximately 4 million transistors, almost half ofwhich are flash memory or static RAM. This leads to an estimated area of8 mm². The image processor contains:

-   -   a CPU/microcontroller core;    -   program storage memory, which is suitably 8 Kbytes of flash        memory;    -   program variable storage, which is suitably 2 KByte of RAM;    -   image processing units, including an image capture unit, image        histogram unit and an image enhancement unit;    -   a Serial Bus Interface;    -   a parallel interface;

The image processor may also include a number of housekeeping andadministration elements including;

-   -   image storage RAM, which is suitably multi-level Flash memory;    -   a memory decoder;    -   a clock; and    -   a joint test action group (JTAG) interface.

The image processor is intended to run at a clock speed of approximately24 MHz on 3V externally and 1.5V internally to minimize powerconsumption. The actual operating frequency will be an integer multipleof the Serial Bus operating frequency. The CPU is intended to be asimple micro-controller style CPU, running at about 1 MHz, and can be avendor supplied core.

The image processor incorporates a simple micro-controller CPU core 171to synchronize image capture and printing image processing chains and toperform general operating system duties. A wide variety of CPU cores aresuitable, it can be any processor core with sufficient processing powerto perform the required calculations and control functions fast enoughto meet consumer expectations.

Since all of the image processing is performed by dedicated hardware,the CPU does not have to process pixels. As a result, the CPU can beextremely simple. An example of a suitable core is a Philips 8051micro-controller running at about 1 MHz.

Associated with the CPU Core 171 is a Program ROM 172 and a smallProgram Scratch RAM 173. The CPU 171 communicates with the other unitswithin the image processor via memory-mapped I/O supported by a MemoryDecoder 174. Particular address ranges map to particular units, andwithin each range, to particular registers within that particular unit.This includes the serial interface 176 and parallel interface 177described below.

The CPU Memory Decoder 174 is a simple decoder for satisfying CPU dataaccesses. The Decoder translates data addresses into internal registeraccesses over the internal low speed bus 175, and therefore allows formemory mapped I/O of image processor registers. The bus 175 includesaddress lines 175 a and data or control lines 175 b.

The small Program Flash ROM 172 is incorporated into the image processorto store simple sequences for controlling the operation of the imageprocessor 170. The ROM size depends on the CPU chosen, but should not bemore than 8 Kbytes.

Likewise, a small scratch RAM 173 is incorporated into the imageprocessor for, primarily, program variable storage. Since the programcode does not have to manipulate images, there is no need for a largescratch area. The RAM size depends on the CPU chosen (e.g. stackmechanisms, subroutine calling conventions, register sizes etc.), butshould not be more than about 2 Kbytes

The Serial Bus interface 176, is connected to the internal chiplow-speed bus 175. The Serial Bus is controlled by the CPU 171 andpreferably follows the USB protocol, although other protocols may besuitable. The maximum speed on the bus is 12 MBits/sec, although themaximum effective data transfer rate is 8 MBits/sec due to protocoloverhead and transmission redundancy. The Serial Bus is used to transmitcommands and images between the various modules of the compact printersystem and is described in a co-pending application titled ModularCompact Printer System.

Since the image processor 170 is responsible for transmitting an imageto the Printer Module 10, the transmission timing considerations arerequired for image processing parameters. The time taken to transmit acomplete image (850×534 L*a*b*) is 1.36 seconds (850×534×3×8/8,000,000).

The parallel interface 177 connects the image processor to individualstatic electrical signals, such as LCD segments 186 and buttons 187 (eg.self timer). The CPU 171 is able to control each of these connections asmemory-mapped I/O via the low-speed bus 175. The following table showsthe connections to the parallel interface.

Connections to Parallel Interface Connection Direction Pins “Take”button In 1 Timer on/off switch In 1

A standard JTAG (Joint Test Action Group) Interface 183 may be includedin the image processor for testing purposes. Due to the complexity ofthe chip, a variety of testing techniques are required, including BIST(Built In Self Test) and functional block isolation. An overhead of 10%in chip area is assumed for overall chip testing circuitry.

The image processor 170 may also include a clock phase-locked loop 184that provides timing signals to the controller. The clock 184 draws abase signal from crystal oscillator 185. Some CPU include a clock so theclock 184 and crystal 185 may not be required.

An Image Storage Memory 181 is used to store a captured image obtainedfrom the image sensor 182 via the image capture unit 178. It is suitablyanalog multi-level Flash RAM (4-bits per cell) so that the image isretained after the power has been shut off. The Image Storage Memory isreferred to as ImageRAM for convenience Although Flash memory is notrequired, multilevel Flash memory requires fewer gates than RAM, and theuse of 16-level Flash (4-bit per cell) is possible as the occasional biterror in the image is not fatal (compared to program code).

The image is written to ImageRAM 181 by the Image Capture Unit 178, andread by image processing units which are suitably an Image HistogramUnit 179 and the Image Enhancement Unit 180. The Image Histogram Unit179 and the Image Enhancement Unit 180 could be embodied in a singleimage processing unit but the inventors find that it is convenientconceptually to treat the processes performed by each unit separately.The CPU 171 does not have direct random access to this image memory.

The Image Capture Unit 178 accepts pixel data from the image sensor 182via an Image Sensor Interface 188, linearizes the RGB data via a lookuptable 189, and finally writes the linearized RGB image to ImageRAM 181in planar format. The process is shown in FIG. 18.

The total amount of memory required for the planarized linear RGB imageis 500,000 bytes (approximately 0.5 MB) arranged as follows:

R: 425×267=113,475 bytes

B: 425×267=113,475 bytes

G: 425×537=226,950 bytes

The Image Sensor Interface (ISI) 188 is a state machine that sendscontrol information to the external CMOS Image Sensor 182, includingframe sync pulses and pixel clock pulses in order to read the image. TheISI may be a sourced cell from the image sensor manufacturer. The ISI iscontrolled by an Image Capture Unit State Machine 190.

Although a variety of image sensors are available, only the Bayer colorfilter array (CFA) is considered here by way of example. Persons skilledin the art will be able to readily extend the invention to other imagesensor formats.

The image captured by the CMOS sensor (via a taking lens) is assumed tohave been sufficiently filtered so as to remove any aliasing artifacts.The sensor itself has an aspect ratio of approximately 3:2, with aresolution of 850×534 samples. The most likely pixel arrangement is theBayer color filter array (CFA), with each 2×2 pixel block arranged in amosaic of one red pixel, one blue pixel, and two green pixels on adiagonal of the pixel block.

Each contone sample of R, G, or B (corresponding to red, green, and bluerespectively) is 10-bits. Note that each pixel of the mosaic containsinformation about only one of R, G, or B. Estimates of the missing colorinformation must be made before the image can be printed.

The CFA is considered to perform some amount of fixed pattern noise(FPN) suppression. Additional FPN suppression may required.

The image sensor is unlikely to have a completely linear response.Therefore the 10-bit RGB samples from the CFA must be considered to benon-linear. These non-linear samples are translated into 8-bit linearsamples by means of a lookup table (one table per color). The lookuptable 189 is a ROM mapping the sensor's RGB response to a linear RGB. Assuch, the ROM is 3 KBytes (3×1024×8-bits). Ten bits of address come fromthe ISI, and two bits of TableSelect are generated by the Image CaptureUnit's State Machine 190.

The Image Capture Unit's State Machine 190 generates control signals forthe Image Sensor Interface, and generates addresses for linearizing theRGB and for planarizing the image data.

The control signals sent to the ISI inform the ISI to start capturingpixels, stop capturing pixels etc.

The 2-bit address sent to the Lookup Table matches the current linebeing read from the ISI. For even lines (0, 2, 4 etc.), the 2-bitaddress is Red, Green, Red, Green etc. For odd lines (1, 3, 5 etc.), the2-bit address is Green, Blue, Green, Blue. This is true regardless ofthe orientation of the camera.

A 21-bit address is sent to the Image RAM as the write address for theimage. Three registers hold the current address for each of the red,green, and blue planes. The addresses increment as pixels are written toeach plane.

The Image Capture Unit contains a number of registers which aresummarized in the following table.

Registers in Image Capture Unit Name Bits Description MaxPixels 10Number of pixels each row MaxRows 10 Number of rows of pixels in imageCurrentPixel 10 Pixel currently being fetched CurrentRow 10 Rowcurrently being processed NextR 19 The address in Image RAM to store thenext Red pixel. Set to start address of red plane before image capture.After image capture, this register will point to the byte after the redplane. NextG 19 The address in Image RAM to store the next Green pixel.Set to start address of green plane before image capture. After imagecapture, this register will point to the byte after the green plane.NextB 19 The address in Image RAM to store the next Blue pixel. Set tostart address of blue plane before image capture. After image capture,this register will point to the byte after the blue plane. EvenEven 2Address to use for even rows/even pixels EvenOdd 2 Address to use foreven rows/odd pixels OddEven 2 Address to use for odd rows/even pixelsOddOdd 2 Address to use for odd rows/odd pixels Go 1 Writing a 1 herestarts the capture. Writing a 0 here stops the image capture. A 0 iswritten here automatically by the state machine after MaxRows ofMaxPixels have ben captured.

In addition, the Image Sensor Interface 188 will contain a number ofregisters. The exact number of registers will depend on the Image Sensorchosen.

The Image Histogram Unit (IHU) 179 is designed to generate histogramsfor planar format images with samples of eight bits each. The ImageHistogram Unit is typically used three times per print. Three differenthistograms are gathered, one per color plane. Each time a histogram isgathered, the results are analyzed in order to determine the low andhigh thresholds, scaling factors etc. for use in the remainder of theprint process. Suitable histogram processes will be known to personsskilled in the art.

The histogram itself is stored in a 256-entry RAM, each entry being 18bits. The histogram RAM is only accessed from within the IHU. Individualentries are read from and written to as 18-bit quantities.

The Image Enhancement Unit (IEU) 180 provides the interface between theImage RAM 181 and the Serial Bus Interface 176, as shown in FIG. 19. TheIEU takes a planarized linear RGB obtained from a CFA format capturedimage from the ImageRAM, and produces a fully populated L*a*b* image ofresolution 850×534 for subsequent transmission to the Printer Module 10via the Serial Bus Interface 176.

The image enhancement process includes the following functions:

Gather Statistics;

Rotate Image;

White Balance and Range Expansion;

Resample;

Convert to L*a*b*.

Sharpen;

The IEU 180 performs all of these functions with the exception of GatherStatistics. To perform the Gather Statistics step, the CPU calls theImage Histogram Unit 179 three times (once per color channel), andapplies simple thresholding algorithms.

In terms of speed, the IEU is producing data for the Serial BusInterface. Given that the Serial Bus Interface transmits at an effectiverate of 8 MHz, and the image processor 170 runs at 24 MHz, it takes theserial interface 72 cycles to transmit the 24 bits for each pixel (3colors @ 8 bits per color @ 3 cycles to transfer each bit). The IEUtherefore has 72 cycles to produce each L*a*b* triplet.

The IEU performs the four processes shown in FIG. 19 in parallel. Thefirst process performs White Balance and Range Expansion. The secondprocess performs Resampling. The third performs color conversion, andthe fourth performs sharpening. The output from the IEU is directly sentto the Serial Bus Interface for transmission to the Printer Module. Theprocesses are connected via buffers, each typically only a few bytes.The structure of the IEU is shown in FIG. 19. The buffer sizes aresummarized in the following table.

Buffer sizes for Image Enhancement Unit Size Buffer (bytes) Compositionof Buffer Buffer A 172.5 Red Buffer = 6 lines of 6 entries @ 10-bitseach = 45 bytes Blue Buffer = 6 lines of 6 entries @ 10-bits each = 45bytes Green Buffer = 11 lines of 6 entries @ 10-bits each = 82.5 bytesBuffer B 22.5 2 sets of 3 RGB pixels = 2 × 3 × 3 entries @ 10-bits each= 180 bits Buffer C 20 5 × 4 RAM 3 lines of 4 entries of L @ 8-bits each= 12 bytes 2 colors × 4 entries @ 8-bits each = 8 bytes Buffer D 6 2 L *a * b * pixels = 2 × 3 entries @ 8-bits each = 6 bytes

The task of loading Buffer A 192 from the ImageRAM 181 involves thesteps of white balance and range expansion 191. The pixels must beproduced for Buffer A fast enough for their use by the Resamplingprocess. This means that during a single group of 72 cycles, this unitmust be able to read, process, and store 6 red pixels, 6 blue pixels,and 11 green pixels.

Once a given pixel has been read from the appropriate plane in the imagestore, it must be white balanced and its value adjusted according to therange expansion calculation defined as:

Pixel′=(Pixel−LowThreshold)×RangeScaleFactor

where RangeScaleFactor=256/(HighThreshold−LowThreshold)

The process simply involves a single subtraction (floor 0), and amultiply (255 ceiling), both against color specific constants. Thestructure of this unit is shown in FIG. 20.

The red, green and blue low thresholds 199, together with the red,green, and blue scale factors 200 are determined by the CPU 171 aftergenerating the histograms for each color plane via the Image HistogramUnit.

Depending on whether the current pixel being processed in the pipelineis red, green, or blue, the appropriate low threshold and scale factoris multiplexed into the subtract unit and multiply unit, with the outputwritten to the appropriate color plane in Buffer A.

The Subtract unit subtracts the 8-bit low Threshold value from the 8-bitImage RAM pixel value, and has a floor of 0. The 8-bit result is passedon to the specialized 8×8 multiply unit, which multiplies the 8-bitvalue by the 8-bit scale factor (8 bits of fraction, integer=1). Onlythe top 10 bits of the result are kept, and represent 8 bits of integerand 2 bits of fraction. The multiplier has a result ceiling of 255, soif any bit higher than bit 7 would have been set as a result of themultiply, the entire 8-bit integer result is set to 1s, and thefractional part set to 0.

Apart from the subtraction and multiply units, the majority of work inthis unit is performed by the Address Generator 201, which iseffectively the state machine for the unit. The address generation isgoverned by two factors: on a given cycle, only one access can be madeto the Image RAM, and on a given cycle, only one access can be made toBuffer A. Of the 72 available cycles, 3 sets of 16 cycles are used bythe Resampler for reading Buffer A. The actual usage is 3 sets of 24cycles, with 16 reads followed by 8 wait cycles. That gives a total of24 available cycles for 23 writes (6 red, 6 blue, 11 green). This meansthe two constraints are satisfied if the timing of the writes to BufferA coincide with the wait cycles of the Resampler.

Buffer A holds the white-balanced and range-expanded pixels at theoriginal capture spatial resolution. Each pixel is stored with 10 bitsof color resolution, compared to the ImageRAM image storage colorresolution of 8 bits per pixel.

Buffer A is arranged as three separately addressable buffers 192 r, 192g, 192 b—one for each color plane of red, green, and blue.

During the course of 72 cycles, 16 entries are read from each of thethree buffers three times by the Resampling process, and up to 29 newvalues are written to the three buffers (the exact number depends on thescale factor and the current sub-pixel position during resampling). Thebuffers must be wide enough so that the reading and writing can occurwithout interfering with one another. During the read process, fourpixels are read from each of six rows. On average each input pixel isused twice.

The green plane has a D value of 0.5 for resampling, indicating thatfour sample positions can be contained within two CFA pixel positions.However, each row of green samples only holds every alternate pixel.This means that only four samples are required per row (worst case is 4,not 3, due to a worst case initial position). Movement in Y indicatesthe requirement of an additional sample column, making five. Finally, anadditional sample column is required for writing. This gives a total ofsix samples per row. Seven rows are required for a single sample. Togenerate the three sets of RGB pixels for each X position, the maximummovement in Y will be two rows. Movement in X adds one sample row aboveand below. Consequently a total of 11 rows are required.

The red and blue planes have a D value of 0.5 for resampling, indicatingthat 4 locations can be contained within two samples. Four samples perrow are required for the resampling process, which is further increasedto six samples to match the green plane (for startup purposes). Six rowsare required to cater for movement in Y.

Each sub-buffer is implemented as a RAM with decoding to read or write asingle 10-bit sample per cycle. The sub-buffers are summarized in thefollowing table, and consume less than 175 bytes.

Sub-Buffer Summary Buffer Composition Bits Red Buffer  6 rows × 6samples × 10-bits 360 Blue Buffer  6 rows × 6 samples × 10-bits 360Green Buffer 11 rows × 6 samples × 10 bits 660

The Resample process is responsible for generating the full complementof R, G, and B pixels for each CFA coordinate by appropriate resamplingof the white-balanced and range-expanded R, G, and B planar images.

The time allowed for producing the components of R, G, and B is 72cycles. However we must effectively produce RGB values for three pixelcoordinates: the pixel in question, and the pixel above and below. Thuswe have 72 cycles in which to calculate three RGB samples.

Buffering RGB values to save recalculation requires too much memory, andin any case, there is sufficient time to generate the RGB values.

The resampling process can be seen as three sets of RGB generation, eachof which must be completed within 24 cycles (for a total maximum elapsedtime of 72 cycles). The process of generating a single RGB value can inturn be seen as three processes performed in parallel: the calculationof R, the calculation of G, and the calculation of B, all for a givenpixel coordinate. This is effectively running three image reconstructionfilters, one on each channel of the image. In the case of the imageprocessor 170 we perform image reconstruction with five sample points,requiring four coefficients in a convolution kernel (since onecoefficient is always 0 and thus the sample point is not required).

Consequently, calculation of the medium resolution R pixel is achievedby running an image reconstruction filter on the R data. Calculation ofthe medium resolution G pixel is achieved by running an imagereconstruction filter on the G data, and calculation of the mediumresolution B pixel is achieved by running an image reconstruction filteron the B data. Although the kernels are symmetric in x and y, they arenot the same for each color plane. R and B are likely to be the samekernel due to their similar image characteristics, but the G plane, dueto the rotation required for image reconstruction, must have a differentkernel. A high level view of the process can be seen in FIG. 21.

The resampling process can only begin when there are enough pixels inBuffer A for the current pixel line being generated. This will be thecase once four columns of data have been written to each of the colorplanes in Buffer A. The Resampling process must stall until that time.

To calculate a given color plane's medium resolution pixel value, wehave 24 cycles available. To apply the kernel to the 4×4 sample area, weapply a 1D kernel (indexed by x) on each of the four rows of four inputsamples. We then apply the 1D kernel (indexed by y) on the resultantfour pixel values. The final result is the output resampled pixel.Applying a single coefficient each cycle gives a total of 16 cycles togenerate the four intermediate values, and four cycles to generate thefinal pixel value, for a total of 20 cycles.

With regards to precision, the input pixels are each 10 bits (8:2), andkernel coefficients are 12 bits. We keep 14 bits of precision during thefour steps of each application of the kernel (8:6), but only save 10bits for the result (8:2). Thus the same convolve engine can be usedwhen convolving in x and y. The final output of R, G, or B is only 8bits.

The process of resampling then, involves 20 cycles, as shown in thefollowing table. Note that Row 1, Pixel 1 etc. refers to the input fromBuffer A, and is taken care of by an Address Generator.

The 20 Cycle Resample Cycle Kernel Apply Kernel to: Store Result in 1X[1] Row 1, Pixel 1 TMP 2 X[2] Row 1, Pixel 2 TMP 3 X[3] Row 1, Pixel 3TMP 4 X[4] Row 1, Pixel 4 TMP, V1 5 X[1] Row 2, Pixel 1 TMP 6 X[2] Row2, Pixel 2 TMP 7 X[3] Row 2, Pixel 3 TMP 8 X[4] Row 2, Pixel 4 TMP, V2 9X[1] Row 3, Pixel 1 TMP 10 X[2] Row 3, Pixel 2 TMP 11 X[3] Row 3, Pixel3 TMP 12 X[4] Row 3, Pixel 4 TMP, V3 13 X[1] Row 4, Pixel 1 TMP 14 X[2]Row 4, Pixel 2 TMP 15 X[3] Row 4, Pixel 3 TMP 16 X[4] Row 4, Pixel 4TMP, V4 17 Y[1] V1 TMP 18 Y[2] V2 TMP 19 Y[3] V3 TMP 20 Y[4] V4 TMP (foroutput)

Buffer B 194 holds three complete sets of RGB pixel values (10-bits percolor component) for a given CFA coordinate in a double buffered format(one set of 3 RGB pixels is being read by the Color Convert process,while the other is being written to 10-bits at a time by the Resampleprocess).

The values are moved from the first RGB buffer to the second once theColor Convert and Resample processes have both finished. This can simplybe achieved by a Select bit that is toggled, rather than physicallytransferring the data from one set of 9 bytes to the other.

The color conversion process 195 must produce contone L*a*b* pixels forthe Sharpen process within 72 cycles. Since the sharpening process onlyrequires the L* values corresponding to the first and third RGB sets,and only requires the full L*a*b* set for the second RGB set, we have 72cycles in which to perform 5 color conversions (three sets of RGB to L*,and 1 set each of RGB to a* and RGB to b*).

The process requires 14 cycles per color component, leading to a totalof 70 cycles for five conversions (leaving 2 cycles spare).

The conversion is performed as tri-linear interpolation described inco-pending application titled Color Conversion Method for CompactPrinter System. Three 17×17×17×8-bit lookup tables 202 are used for theconversion process: RGB to L*, RGB to a*, and RGB to b*.

Address generation for indexing into the lookup tables isstraightforward. We use the four most significant bits of each 8-bitcolor component for address generation, and the four least significantbits of each 8-bit color component for interpolating between valuesretrieved from the conversion tables. The addressing into the lookuptable requires an adder due to the fact that the lookup table hasdimensions of 17 rather than 16. Fortunately, multiplying a 4-bit numberX by 17 is an 8-bit number XX, and therefore does not require an adderor multiplier, and multiplying a 4-bit number by 17² (289) is onlyslightly more complicated, requiring a single add.

Although the interpolation could be performed faster, a single adder isused to generate addresses and have a single cycle interpolation unit.Consequently it is possible to calculate the interpolation forgenerating a single color component from RGB in 14 cycles ininterpolator 203. The process must be repeated five times, once for eachcolor conversion. Faster methods are possible, but not necessary.

A block diagram of the color conversion process is shown in FIG. 22. Thebasic 14 cycle color conversion process is run five times as follows:

on RGB1 to generate L*1

on RGB2 to generate L*2

on RGB3 to generate L*3

on RGB2 to generate a*

on RGB2 to generate b*

Address generation for writing to Buffer C makes use of the cyclicalnature of Buffer C. The address consists of a 2-bit column component(representing which of the four columns should be written to), and a3-bit value representing L*1, L*2, L*3, a*, or b*. The column numberstarts at 0 each new line and increments (with wrapping) every 72cycles.

Buffer C 196 accepts the output from the Color Convert process 195,where a complete L*a*b* pixel is generated from the RGB equivalent for agiven pixel coordinate. Buffer C is used by the Sharpen process 197,which requires a 3×3 set of luminance values centered on the pixel beingsharpened.

Consequently, during the sharpening process, there is need for access tothe 3×3 array of luminance values, as well as the corresponding a*b*value for the center luminance pixel. At the same time, the next threeL*a*b* values must be calculated from the RGB values by the ColorConvert process.

The actual implementation of Buffer C is simply as a 4×5 (20 entry)8-bit RAM, with the addressing on read and write providing the effectiveshifting of values. A 2-bit column counter can be incremented withwrapping to provide a cyclical buffer, which effectively implements theequivalent of shifting the entire buffer's data by 1 column position.The fact that the fourth column of a*b* data is not required is notrelevant, and merely uses 2 bytes at the saving of not having toimplement complicated shift and read/write logic. In a given cycle, theRAM can either be written to or read from. The read and write processeshave 72 cycles in which to complete in order to keep up with thetransmission process.

The Sharpen Unit 197 performs a sharpening process which involves ahighpass filter of the luminance channel of the image (the L* channel).The highpass filter used is a basic highpass filter using a 3×3convolution kernel.

The high pass filter is calculated over ten cycles. The first cycleloads the temporary register with eight times the center pixel value(the center pixel shifted left by 3 bits). The next eight cyclessubtract the remaining eight pixel values, with a floor of 0. Thus theentire procedure can be accomplished by an adder. Cycle 10 involves themultiplication of the result by a constant. This constant is therepresentation of 1/9, but is a register to allow the amount to bealtered by software by some scale factor.

The resultant sharpened L* is written out to Buffer D 198 during cycle11, and the a*, and b* color components are copied to Buffer D duringcycles 12 and 13. The structure of the Sharpen unit can be seen in FIG.23.

The adder unit 204 connected to Buffer C is a subtractor with a floor of0. TMP 205 is loaded with 8× the first L value during cycle 0 (of 75),and then the next 8 L values are subtracted from it. The result is notsigned, since the subtraction has a floor of 0.

During the 10th cycle (Cycle 9), the 11 bit total in TMP is multipliedby a scale factor 206 (typically 1/9, but under software control so thatthe factor can be adjusted) and written back to TMP. Only 8 integer bitsof the result are written to TMP (the fraction is truncated), so thelimit from the multiply unit is 255. If a scale factor of 1/9 is used,the maximum value written will be 226 (255× 8/9). The scale factor is 8bits of fraction, with the high bit representing ⅛.

Address Generation 207 is straightforward. Writing to Buffer D is simplyL*, a*, and b* in cycles 11, 12, and 13 respectively. Reading fromBuffer C makes use of the cyclical nature of Buffer C. The addressconsists of a 2-bit column component (representing which of the fourcolumns should be read), and a 3-bit value representing L*1, L*2, L*3,a*, or b*. The column number starts at 1 each line and increments (withwrapping) every 72 cycles.

Sharpening can only begin when there have been sufficient L*a*b* pixelswritten to Buffer C (so that the highpass filter is valid). The sharpenprocess must therefore stall until the color conversion process haswritten three columns of data to Buffer C.

Buffer D 198 holds a complete L*a*b* pixel value (8-bits per colorcomponent) ready to be transmitted by the Serial Bus Interface 176 tothe Printer Module 10. Buffer D is a double buffered format (one L*a*b*pixel is being read by the Serial Bus Interface process, while the otheris being written to by the Sharpen process during a 72 cycle period).

The values are moved from the first buffer to the second once the SerialBus Interface has transmitted all 24 bits of L*a*b*. As with Buffer B,the double buffering can simply be achieved by a Select bit that istoggled, rather than physically transferring the data from one set of 3bytes to the other. The Serial Bus Interface supplies the WriteEnablesignal for the transfer.

It will be appreciated that the processes described herein are simpleoperations on each value stored in an array of values. Addressgeneration satisfies the requirements of accessing array locations inthe appropriate order for reading and writing.

Throughout the specification the aim has been to describe the preferredembodiments of the invention without limiting the invention to any oneembodiment or specific collection of features. Persons skilled in therelevant art may realize variations from the specific embodiments thatwill nonetheless fall within the scope of the invention.

1. A camera module comprising: an image sensor for capturing an image;image storage memory for storing said image; image processing units,being application specific integrated circuits, in communication withsaid image storage memory to transform said image into a deviceindependent color space; and a serial bus interface for communicatingsaid transformed image with a serial bus of a printer moduleinterconnected with the camera module.
 2. The camera module of claim 1wherein the device independent color space is the CIE L*a*b* colorspace.
 3. The camera module of claim 1 wherein said serial bus interfaceis in communication with one or more further modules, said Serial Buscommunicating power and data between said camera module and said one ormore further modules.
 4. The camera module of claim 1, wherein thecamera module receives operational power only from the printer modulevia the Serial Bus.
 5. The camera module of claim 1 wherein said imagestorage memory is for storing a single image.